Add Delta Lake 4.0 support and consolidate common code across delta-33x and delta-40x - #13570

Fixes #13339.

Description

This PR adds GPU acceleration support for Delta Lake 4.0.x and consolidates common code between Delta 3.3.x and 4.0.x implementations. By extracting shared logic into base classes and using a shims pattern for version-specific APIs, we reduce code duplication while maintaining compatibility across both Delta versions.

Key Changes

1. Delta Lake 4.0.x Support

• New Module: Added delta-40x module with full Delta Lake 4.0 compatibility
• Spark 4.0 APIs: Adapted to Spark 4.0's new session hierarchy (SqlSparkSession, ClassicSparkSession)
• Expression Handling: Added support for Spark 4.0's RuntimeReplaceable expressions in statistics collection

2. Code Consolidation Strategy

Shared Base Classes (delta-lake/common/src/main/delta-33x-40x/scala/)

Command Base Classes:

• GpuDeleteCommandBase - Consolidates DELETE command logic
• GpuUpdateCommandBase - Consolidates UPDATE command logic
• GpuCreateDeltaTableCommandBase - Unifies table creation flow for table operations
• GpuOptimizeTableCommand - Common OPTIMIZE command implementation

Catalog & Data Source:

• GpuDeltaCatalogBase - Shared catalog plugin implementation for creating/managing Delta tables
• DeltaProviderBase - Common Delta data source provider logic
• GpuDeltaParquetFileFormatCommon - Unified Parquet format handling with deletion vectors, column mapping, and predicate pushdown

GPU Overrides Metadata:

• DeleteCommandMetaBase, UpdateCommandMetaBase - Shared metadata for command GPU fallback rules
• MergeIntoCommandMetaBase, OptimizeTableCommandMetaBase - Consolidates override metadata logic
• RapidsRowIndexFiltersBase - Common utilities for deletion vector row index filtering

Version Abstraction Layer:

• DeltaCommandShims - Trait abstracting Spark API differences (session types, DataFrame/Column creation, caching)
• DeltaRuntimeShimBase - Runtime utilities for Delta-specific operations
• GpuDeltaFileFormatWriterCommon - Shared file writing infrastructure
• MetadataShims - Metadata conversion and compatibility utilities

Version-Specific Implementations

Delta 3.3.x (Delta33xCommandShims):

• Uses standard SparkSession throughout
• Dataset.ofRows() for DataFrame creation
• Direct new Column(expr) for expression wrapping

Delta 4.0.x (Delta40xCommandShims):

• Uses SqlSparkSession for run() methods, ClassicSparkSession for operations
• TrampolineConnectShims.createDataFrame() for Classic/Connect compatibility
• DFUDFShims.exprToColumn() for Column creation
• RuntimeReplaceable expression unwrapping for stats collection

3. What Remains Version-Specific

The following components have too many version-specific differences to consolidate. We can try to consolidate these files in follow-on PR's:

• GpuOptimisticTransaction - Complex transaction logic with version-specific stats/write paths
• GpuMergeIntoCommand - Scattered Spark API differences throughout the file
• GpuDeltaFileFormatWriter - Version-specific write implementations

4. Testing

• Spark 3.56 (Scala 2.12, Delta 3.3.x): Full build and verification passing
• Spark 4.00 (Scala 2.13, Delta 4.0.x): Full build and verification passing
• Integration tests updated for Delta 4.0 compatibility. We have disabled ANSI mode in some of the tests. The test was throwing ARITHMETIC_OVERFLOW, CAST errors for the computation. These are already tested in arithmetic_ops.py tests. So disabling ANSI mode in delta tests should be fine.
  Verified that delta all delta integration tests pass on : Delta-3.3.0 (Spark-3.5.6) and Delta-4.0.0(Spark-4.0.0)

Benchmark Run results:

No major regression between Delta-330 and Delta-400.

Checklists

• This PR has added documentation for new or modified features or behaviors.
• This PR has added new tests or modified existing tests to cover new code paths.
  (Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
• Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.